{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "de5bf9b5",
   "metadata": {},
   "source": [
    "## Comp3-3组学(模态)融合 - Ensemble后融合\n",
    "\n",
    "融合一般指的是对一个研究对象使用多种不同的数据，或者相同数据不同模型预测结果刻画结果的综合考虑。一般情况下，融合分为前融合、后融合\n",
    "\n",
    "* 前融合一般是数据层面的融合。\n",
    "* 后融合一般是结果层面的融合。\n",
    "\n",
    "我们分别使用Comp1-1抽取出来的组学特征与蛋白质组学进行前融合，使用`组学预测结果`与`量表数据`进行后融合。后偶融合分为两种：\n",
    "1. ensemble, 一般是最终结果进行融合，融合可以分为软投票和硬投票。\n",
    "2. stacking，一般是结果在使用一个机器学习算法模型进行融合。\n",
    "\n",
    "##### **注意：由于使用后融合技术，需要保持数据筛选的样本一样**\n",
    "\n",
    "## Onekey步骤\n",
    "\n",
    "1. 数据校验，检查数据格式是否正确。\n",
    "3. 查看一些统计信息，检查数据时候存在异常点。\n",
    "4. 正则化，将数据变化到服从 N~(0, 1)。\n",
    "5. 通过相关系数，例如spearman、person等筛选出特征。\n",
    "6. 构建训练集和测试集，这里使用的是随机划分，正常多中心验证，需要大家根据自己的场景构建两份数据。\n",
    "7. 通过Lasso筛选特征，选取其中的非0项作为后续模型的特征。\n",
    "8. 使用机器学习算法，例如LR、SVM、RF等进行任务学习。\n",
    "9. 【增加融合模块】将以上所有算法进行Ensemble融合。\n",
    "    1. average软投票\n",
    "    2. weight average加权投票。\n",
    "9. 模型结果可视化，例如AUC、ROC曲线，混淆矩阵等。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e35ccb4",
   "metadata": {},
   "source": [
    "## 一、数据校验\n",
    "首先需要检查诊断数据，如果显示`检查通过！`择可以正常运行之后的，否则请根据提示调整数据。\n",
    "\n",
    "数据文件中的数据都是数值类型，或者可以映射成数值类型，这里的`label`某些情况下可能是非数值的，需要自定义数值函数。\n",
    "\n",
    "**注意：在使用树模型时，可以存在缺失，但是线性模型不允许缺失，请自行根据需要填充缺省值**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8d25459",
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import namedtuple\n",
    "import onekey_algo.custom.components as okcomp\n",
    "from onekey_algo import OnekeyDS as okds\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# 读取数据，B超诊断阳性=1，bc_data.csv是要读取的数据。\n",
    "data_file = okds.grade\n",
    "labels = ['B超诊断阳性=1']\n",
    "\n",
    "structed_data = pd.read_csv(data_file, header=0)\n",
    "structed_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e725e41",
   "metadata": {},
   "source": [
    "### 特征维度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a01ebd89",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 删掉ID这一列。\n",
    "structed_data = structed_data.drop(['ID'], axis=1)\n",
    "structed_data.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "214cd32a",
   "metadata": {},
   "source": [
    "## 二、数据统计\n",
    "\n",
    "1. count，统计样本个数。\n",
    "2. mean、std, 对应特征的均值、方差\n",
    "3. min, 25%, 50%, 75%, max，对应特征的最小值，25,50,75分位数，最大值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8141707",
   "metadata": {},
   "outputs": [],
   "source": [
    "structed_data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "373d90df",
   "metadata": {},
   "source": [
    "## 三、正则化\n",
    "\n",
    "`normalize_df` 为onekey中正则化的API，将数据变化到0均值1方差。正则化的方法为\n",
    "\n",
    "$column = \\frac{column - mean}{std}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67c7aebe",
   "metadata": {},
   "outputs": [],
   "source": [
    "from onekey_algo.custom.components.comp1 import normalize_df\n",
    "data = normalize_df(structed_data, not_norm=labels)\n",
    "data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9713a9d6",
   "metadata": {},
   "source": [
    "## 四、相关系数\n",
    "\n",
    "计算相关系数的方法有3种可供选择\n",
    "1. pearson （皮尔逊相关系数）: standard correlation coefficient\n",
    "\n",
    "2. kendall (肯德尔相关性系数) : Kendall Tau correlation coefficient\n",
    "\n",
    "3. spearman (斯皮尔曼相关性系数): Spearman rank correlation\n",
    "\n",
    "三种相关系数参考：https://blog.csdn.net/zmqsdu9001/article/details/82840332"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8956b51",
   "metadata": {},
   "outputs": [],
   "source": [
    "pearson_corr = data.corr('pearson')\n",
    "kendall_corr = data.corr('kendall')\n",
    "spearman_corr = data.corr('spearman')\n",
    "\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from onekey_algo.custom.components.comp1 import draw_matrix\n",
    "plt.figure(figsize=(15.0, 12.0))\n",
    "\n",
    "# 选择可视化的相关系数\n",
    "draw_matrix(pearson_corr, annot=True, cmap='YlGnBu', cbar=False)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "583789c9",
   "metadata": {},
   "source": [
    "### 特征筛选\n",
    "\n",
    "根据相关系数，对于相关性比较高的特征（一般文献取corr>0.9），两者保留其一。\n",
    "\n",
    "```python\n",
    "def select_feature(corr, threshold: float = 0.9, keep: int = 1):\n",
    "    \"\"\"\n",
    "    * corr, 相关系数矩阵。\n",
    "    * threshold，筛选的相关系数的阈值，大于阈值的两者保留其一（可以根据keep修改，可以是其二...）。默认阈值为0.9\n",
    "    * keep，可以选择大于相关系数，保留几个，默认只保留一个。\n",
    "    \"\"\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8440cfb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from onekey_algo.custom.components.comp1 import select_feature\n",
    "sel_feature = select_feature(spearman_corr)\n",
    "sel_feature"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88710c9f",
   "metadata": {},
   "source": [
    "### 过滤特征\n",
    "\n",
    "通过`sel_feature`过滤出筛选出来的特征。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf7ad242",
   "metadata": {},
   "outputs": [],
   "source": [
    "sel_data = data[sel_feature]\n",
    "sel_data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7971b1b2",
   "metadata": {},
   "source": [
    "## 五、构建数据\n",
    "\n",
    "将样本的训练数据X与监督信息y分离出来，并且对训练数据进行划分，一般的划分原则为80%-20%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "249fbd78",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import onekey_algo.custom.components as okcomp\n",
    "\n",
    "n_classes = 2\n",
    "y_data = data[labels]\n",
    "X_data = data.drop(labels, axis=1)\n",
    "\n",
    "X_train, X_test, y_train, y_test = okcomp.comp1.split_dataset(X_data, y_data, test_size=0.3)\n",
    "print(f\"训练集样本数：{X_train.shape}, 验证集样本数：{X_test.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8434607c",
   "metadata": {},
   "source": [
    "### Lasso\n",
    "\n",
    "初始化Lasso模型，alpha为惩罚系数。具体的参数文档可以参考：[文档](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Lasso.html?highlight=lasso#sklearn.linear_model.Lasso)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "379f283b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import linear_model\n",
    "\n",
    "models = []\n",
    "for label in labels:\n",
    "    clf = linear_model.Lasso(alpha=0.05)\n",
    "    clf.fit(X_train, y_train[label])\n",
    "    models.append(clf)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "259a9f59",
   "metadata": {},
   "source": [
    "### 特征筛选\n",
    "\n",
    "筛选出其中coef > 0的特征。并且打印出相应的公式。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "706f3f6b",
   "metadata": {},
   "outputs": [],
   "source": [
    "COEF_THRESHOLD = 1e-6 # 筛选的特征阈值\n",
    "scores = []\n",
    "selected_features = []\n",
    "for label, model in zip(labels, models):\n",
    "    feat_coef = [(feat_name, coef) for feat_name, coef in zip(data.columns[:-1], model.coef_) \n",
    "                 if COEF_THRESHOLD is None or abs(coef) > COEF_THRESHOLD]\n",
    "    selected_features.append(feat_coef)\n",
    "    formula = ' '.join([f\"{coef:+.6f} * {feat_name}\" for feat_name, coef in feat_coef])\n",
    "    score = f\"{label} = {model.intercept_} {'+' if formula[0] != '-' else ''} {formula}\"\n",
    "    scores.append(score)\n",
    "    \n",
    "print(scores[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45651757",
   "metadata": {},
   "source": [
    "## 六、模型筛选\n",
    "\n",
    "根据筛选出来的数据，做模型的初步选择。当前主要使用到的是Onekey中的\n",
    "\n",
    "1. SVM，支持向量机，引用参考。\n",
    "2. KNN，K紧邻，引用参考。\n",
    "3. Decision Tree，决策树，引用参考。\n",
    "4. Random Forests, 随机森林，引用参考。\n",
    "5. XGBoost, bosting方法。引用参考。\n",
    "6. LightGBM, bosting方法，引用参考。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43ec9f22",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_names = ['SVM', 'KNN', 'DecisionTree', 'RandomForest', 'ExtraTrees', 'LightGBM', 'XGBoost']\n",
    "models = okcomp.comp1.create_clf_model(model_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "340a8492",
   "metadata": {},
   "source": [
    "### 交叉验证\n",
    "\n",
    "`n_trails`指定随机次数，每次采用的是80%训练，随机20%进行测试，找到最好的模型，以及对应的最好的数据划分。\n",
    "\n",
    "这里的数据并没有使用前面`Lasso`筛选出来的特征进行训练，理论来说，特征筛选仅对线性模型有一定作用，例如`SVM`、`LR`，但是对树模型没什么作用，例如`DecisionTree`、`Random`这些。所以默认不筛选。\n",
    "\n",
    "**注意：这里采用了【挑数据】，如果想要严谨，请修改`n_trails=1`。**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a35cd89",
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "# 随机使用n_trails次数据划分，找到最好的一次划分方法，并且保存在results中。\n",
    "results = okcomp.comp1.get_bst_split(X_data, y_data, models, test_size=0.2, metric_fn=accuracy_score, n_trails=5, cv=True, random_state=0)\n",
    "_, (X_train_sel, X_test_sel, y_train_sel, y_test_sel) = results['results'][results['max_idx']]\n",
    "trails, _ = zip(*results['results'])\n",
    "cv_results = pd.DataFrame(trails, columns=model_names)\n",
    "# 可视化每个模型在不同的数据划分中的效果。\n",
    "sns.boxplot(data=cv_results)\n",
    "plt.ylabel('Accuracy %')\n",
    "plt.xlabel('Model Nmae')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed341946",
   "metadata": {},
   "source": [
    "### 模型筛选\n",
    "\n",
    "使用最好的数据划分，进行后续的模型研究。\n",
    "\n",
    "**注意**: 一般情况下论文使用的是随机划分的数据，但也有些论文使用【刻意】筛选的数据划分。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92bd851d",
   "metadata": {},
   "outputs": [],
   "source": [
    "targets = []\n",
    "for l in labels:\n",
    "    new_models = okcomp.comp1.create_clf_model(model_names).values()\n",
    "    for m in new_models:\n",
    "        m.fit(X_train_sel, y_train_sel[l])\n",
    "        y_pred = m.predict(X_test_sel)\n",
    "    targets.append(new_models)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "403b4286",
   "metadata": {},
   "source": [
    "## 七、预测结果\n",
    "\n",
    "* predictions，二维数据，每个label对应的每个模型的预测结果。\n",
    "* pred_scores，二维数据，每个label对应的每个模型的预测概率值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78c2f702",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score, roc_curve, auc\n",
    "from onekey_algo.custom.components.metrics import analysis_pred_binary\n",
    "\n",
    "predictions = [[model.predict(X_test_sel) for model in target] for label, target in zip(labels, targets)]\n",
    "pred_scores = [[model.predict_proba(X_test_sel) for model in target] for label, target in zip(labels, targets)]\n",
    "\n",
    "metric = []\n",
    "for label, prediction, scores in zip(labels, predictions, pred_scores):\n",
    "    for mname, pred, score in zip(model_names, prediction, scores):\n",
    "        acc, auc, ci, tpr, tnr, ppv, npv, precision, recall, f1, thres = analysis_pred_binary(y_test_sel[label], score[:, 1])\n",
    "        ci = f\"{ci[0]:.4f} - {ci[1]:.4f}\"\n",
    "        metric.append((mname, acc, auc, ci, tpr, tnr, ppv, npv, precision, recall, f1, thres, f\"{mname}\"))\n",
    "metric = pd.DataFrame(metric, index=None, columns=['model_name', 'Accuracy', 'AUC', '95% CI', 'Sensitivity', 'Specificity', \n",
    "                                                   'PPV', 'NPV', 'Precision', 'Recall', 'F1', 'Threshold', 'Task'])\n",
    "metric"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d74b280a",
   "metadata": {},
   "source": [
    "### Average Ensemble\n",
    "\n",
    "Avgerage软投票机制"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce0e5761",
   "metadata": {},
   "outputs": [],
   "source": [
    "fusion_scores = [[np.mean(fusion_score, axis=0, keepdims=False) \n",
    "                 for fusion_score in [np.array(scores)] for scores in pred_scores]]\n",
    "fusion_pred = [[np.argmax(score, axis=1) for score in scores] for scores in fusion_scores]\n",
    "\n",
    "avg_fusion_preddictions = [p + f for p, f in zip(predictions, fusion_pred)]\n",
    "avg_fusion_pred_scores = [p + f for p, f in zip(pred_scores,fusion_scores)]\n",
    "\n",
    "avg_fusion_metric = []\n",
    "avg_fusion_model_names = model_names + ['AvgFusion']\n",
    "for label, prediction, scores in zip(labels, avg_fusion_preddictions, avg_fusion_pred_scores):\n",
    "    for mname, pred, score in zip(avg_fusion_model_names, prediction, scores):\n",
    "        acc, auc, ci, tpr, tnr, ppv, npv, precision, recall, f1, thres = analysis_pred_binary(y_test_sel[label], score[:, 1])\n",
    "        ci = f\"{ci[0]:.4f} - {ci[1]:.4f}\"\n",
    "        avg_fusion_metric.append((mname, acc, auc, ci, tpr, tnr, ppv, npv, precision, recall, f1, thres, f\"{mname}\"))\n",
    "        \n",
    "pd.DataFrame(avg_fusion_metric, index=None, columns=['model_name', 'Accuracy', 'AUC', '95% CI', 'Sensitivity', 'Specificity', \n",
    "                                                     'PPV', 'NPV', 'Precision', 'Recall', 'F1', 'Threshold', 'Task'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c42663f7",
   "metadata": {},
   "source": [
    "### Weighted Average Ensemble\n",
    "\n",
    "Weighted average软投票，基于准确率加权投票。使用准确率的sotfmax归一化进行加权，准确率越高，权重越大。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f308a2ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "def softmax(z):\n",
    "    t = np.exp(z)\n",
    "    a = np.exp(z) / np.sum(t, axis=1)\n",
    "    return a\n",
    "\n",
    "weights = softmax(np.reshape(np.array(metric['Accuracy']), (-1, len(model_names))))\n",
    "\n",
    "fusion_scores = [[np.mean(fusion_score, axis=0, keepdims=False) \n",
    "                  for fusion_score in [np.array([s * w for s, w in zip(scores, weight)])]]\n",
    "                 for weight, scores in zip(weights, pred_scores)]\n",
    "fusion_pred = [[np.argmax(score, axis=1) for score in scores] for scores in fusion_scores]\n",
    "\n",
    "weight_fusion_preddictions = [p + f for p, f in zip(avg_fusion_preddictions, fusion_pred)]\n",
    "weight_fusion_pred_scores = [p + f for p, f in zip(avg_fusion_pred_scores, fusion_scores)]\n",
    "\n",
    "weight_fusion_metric = []\n",
    "weight_fusion_model_names = model_names + ['AvgFusion', 'WeightFusion']\n",
    "for label, prediction, scores in zip(labels, weight_fusion_preddictions, weight_fusion_pred_scores):\n",
    "    for mname, pred, score in zip(weight_fusion_model_names, prediction, scores):\n",
    "        acc, auc, ci, tpr, tnr, ppv, npv, precision, recall, f1, thres = analysis_pred_binary(y_test_sel[label], score[:, 1])\n",
    "        ci = f\"{ci[0]:.4f} - {ci[1]:.4f}\"\n",
    "        weight_fusion_metric.append((mname, acc, auc, ci, tpr, tnr, ppv, npv, precision, recall, f1, thres, f\"Test\"))\n",
    "\n",
    "weight_fusion_metric = pd.DataFrame(weight_fusion_metric, index=None, columns=['model_name', 'Accuracy', 'AUC', '95% CI', 'Sensitivity', \n",
    "                                                                               'Specificity', \n",
    "                                                                               'PPV', 'NPV', 'Precision', 'Recall', 'F1', 'Threshold', \n",
    "                                                                               'Task'])\n",
    "weight_fusion_metric"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c723c08f",
   "metadata": {},
   "source": [
    "### 绘制曲线\n",
    "\n",
    "绘制的不同模型的准确率柱状图和折线图曲线。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7664fdef",
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "sns.set_theme(style=\"whitegrid\")\n",
    "\n",
    "# 设置绘制参数\n",
    "plt.figure(figsize=(10, 10))\n",
    "plt.subplot(211)\n",
    "sns.barplot(x='model_name', y='Accuracy', data=weight_fusion_metric, hue='Task')\n",
    "plt.subplot(212)\n",
    "sns.lineplot(x='model_name', y='Accuracy', data=weight_fusion_metric, hue='Task')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c752e1c",
   "metadata": {},
   "source": [
    "## 绘制ROC曲线\n",
    "确定最好的模型，并且绘制曲线。\n",
    "\n",
    "```python\n",
    "def draw_roc(y_test, y_score, title='ROC', labels=None):\n",
    "```\n",
    "\n",
    "`sel_model = ['AvgFusion', 'WeightFusion']`参数为想要绘制的模型对应的参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d0fe1ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "sel_model = ['AvgFusion', 'WeightFusion']\n",
    "for sm in sel_model:\n",
    "    if sm in weight_fusion_model_names:\n",
    "        sel_model_idx = weight_fusion_model_names.index(sm)\n",
    "    \n",
    "        # Plot all ROC curves\n",
    "        plt.figure(figsize=(8, 8))\n",
    "        okcomp.comp1.draw_roc([np.array(y_test_sel[l]) for l in labels], \n",
    "                              [s[sel_model_idx] for s in weight_fusion_pred_scores], labels=labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6a8438d",
   "metadata": {},
   "source": [
    "## 绘制混淆矩阵\n",
    "\n",
    "绘制混淆矩阵，[混淆矩阵解释](https://baike.baidu.com/item/%E6%B7%B7%E6%B7%86%E7%9F%A9%E9%98%B5/10087822?fr=aladdin)\n",
    "`sel_model = ['SVM', 'KNN']`参数为想要绘制的模型对应的参数。\n",
    "\n",
    "如果需要修改标签到名称的映射，修改`class_mapping={1:'1', 0:'0'}`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0636fca9",
   "metadata": {},
   "outputs": [],
   "source": [
    "sel_model = ['AvgFusion', 'WeightFusion']\n",
    "c_matrix = {}\n",
    "\n",
    "for sm in sel_model:\n",
    "    if sm in weight_fusion_model_names:\n",
    "        sel_model_idx = weight_fusion_model_names.index(sm)\n",
    "        for idx, label in enumerate(labels):\n",
    "            cm = okcomp.comp1.calc_confusion_matrix(weight_fusion_preddictions[idx][sel_model_idx], y_test_sel[label],\n",
    "                                                    class_mapping={1:'1', 0:'0'}, num_classes=2)\n",
    "            c_matrix[label] = cm\n",
    "            plt.figure(figsize=(5, 4))\n",
    "            plt.title(f'Model:{sm}')\n",
    "            okcomp.comp1.draw_matrix(cm, norm=True, annot=True, cmap='Blues')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8210817",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
